Habitat degradation and loss can result in population decline and genetic erosion, limiting the ability of organisms to cope with environmental change, whether this is through evolutionary genetic response (requiring genetic variation) or through phenotypic plasticity (i.e., the ability of a given genotype to express a variable phenotype across environments). Here we address the question whether plants from small populations are less plastic or more susceptible to environmental stress than plants from large populations. We collected seed families from small (<100) versus large natural populations (>1000 flowering plants) of the rare, endemic plant Cochlearia bavarica (Brassicaceae). We exposed the seedlings to a range of environments, created by manipulating water supply and light intensity in a 2 × 2 factorial design in the greenhouse. We monitored plant growth and survival for 300 days. Significant effects of offspring environment on offspring characters demonstrated that there is phenotypic plasticity in the responses to environmental stress in this species. Significant effects of population size group, but mainly of population identity within the population size groups, and of maternal plant identity within populations indicated variation due to genetic (plus potentially maternal) variation for offspring traits. The environment × maternal plant identity interaction was rarely significant, providing little evidence for genetically- (plus potentially maternally-) based variation in plasticity within populations. However, significant environment × population-size-group and environment × population-identity interactions suggested that populations differed in the amount of plasticity, the mean amount being smaller in small populations than in large populations. Whereas on day 210 the differences between small and large populations were largest in the environment in which plants grew biggest (i.e., under benign conditions), on day 270 the difference was largest in stressful environments. These results show that population size and population identity can affect growth and survival differently across environmental stress gradients. Moreover, these effects can themselves be modified by time-dependent variation in the interaction between plants and their environment.